Finding truth even if the crowd is wrong

Over a hundred years ago Galton reported on the uncanny accuracy of the median estimate of the weight of an ox, as judged by spectators at a country fair [1]. Since then, the notion that the ‘wisdom of the crowd’ is superior to any individual has itself become a piece of crowd wisdom, raising expectations that web-based opinion aggregation might replace expert judgment as a source of policy guidance [2, 3]. However, distilling the best answer from diverse opinions is challenging when most people hold an incorrect view [4]. We propose a method based on a new definition of the best answer: it is the one given by respondents who would be least surprised by the true answer if it were revealed. Since this definition is of interest only when the true answer is unknown, algorithmic implementation is nontrivial. We solve this problem by asking respondents not only to answer the question, but also to predict the distribution of others’ answers. Previously, it was shown that this secondary information can be used to create incentives for honest responding [5]. Here we prove that this information can also be used to identify which answer is the best answer by our new definition. Unlike multi-item analysis [6, 7] or boosting [8], our method can be applied to a unique question. This capability is critical in knowledge domains that lack consensus about which historical precedents might establish experts’ relative track records. Unlike Bayesian models [9, 10, 11, 12, 13] our method does not require user-specified prior probabilities, nor does it require information sharing that might lead to “groupthink” [14]. An experiment demonstrates that the method outperforms algorithms based on democratic or confidence-weighted voting [15, 16, 17, 18, 19]. Imagine that you have no knowledge of U.S. geography, and are confronted with the question